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[57] ABSTRACT 

A picture communication apparatus includes an extracting 
circuit for extracting video data of at least one portion from 
video data inputted thereto, an encoder for respectively 
encoding the extracted video data and the remaining video 
data, and a multiplexer for multiplexing the encoded video 
data. When encoding the video data, predetermined amount 
of codes are allocated to the respective video data. This 
suppresses deterioration in the picture quality when con- 
ducting an image communication via a transmission path 
having a low transmission rate. The transmitted video is 
displayed on a contoured display. 

3 Claims, 10 Drawing Sheets 
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IMAGE SIGNAL ENCODING AND Moreover, in a videophone apparatus and a video confer- 
COMMUNICATING APPARATUS USING ence system, a video signal produced by an imaging appa- 
MEANS FOR EXTRACTING PARTICULAR ratus is encoded to be transmitted via a transmission path 
PORTIONS OF AN OBJECT IMAGE such as an integrated services digital network (ISDN). For 
t* AmrrunirNn hp thr TNVFNTTON 5 sample, for component national television system commit- 
B ACKGROUND OF THE INVENTION tee (NTSQ signals, when the signals are not compressed in 
The present invention relates to an image encoding and transmission data, the transfer speed in terms of bits is 216 
picture communication apparatus, for example a video megabits per second (Mbps) according to the studio stan- 
phone and an image recording apparatus. dards of color television. This leads to a requirement that the 
Conventionally, as communication apparatuses to com- i 0 signals are to be more efficiently encoded to reduce the 
Tniinirate voices and tones, there has been used a video n umb er of bits of transmission data. As the encoding 
telephone facility (A. N. Netravali, B. G. Haskell, "Digital method, there has been primarily employed a method 
Picture", pp. 115-1 19, AT&T Bell Lab. (1988)). The appa- described in pages 793 to 799 of the "Journal of Institute of 
ratus includes a sending facility including an imaging Television Engineers of Japan* 1 , VoL 45, No. 7 (1991). 
apparatus, a voice input device, and an encoder circuit for 15 Namely, mere is basically used a conditional pixel supple- 
encoding images and voices; a receiving facility including a m rating method on the basis of inter-frame estimation or 
decoder for decoding signals of images and voices, a display forecasting in which only mobile portions are transmitted 
including a speaker and CRT; and a conamunication con- suc jj that other encoding methods such as a discrete cosine 
troller for cominiinicating images and voices via a network. transform (DCT) are additionally used in combination with 
In such conventional apparatuses, the contents of an image 20 the conditional pixel supplementing method. Thanks to 
produced by a camera are entirely encoded and transmitted development of such a highly efficient encoding technology, 
via a transmission line, which leads to necessity of trans- videophones and video conference systems using ISDN 
mitring a large amount of data. Consequently, a low-priced lines have been widely introduced to practical uses in 
videophone of a type conducting communication via a business and industrial fields. A communication method in 
low-speed analog communication line has been attended 25 which an image is transformed into codes for transmission 
with a problem in which the picture quality is considerably thereof has been described, for example, in the JP-A-62- 
deteriorated or motion of pictures becomes uncomfortable 120179 and in "Systematic Image Encoding" written by 
and unnatural Makoto Miyahara in pages 1 to 14 of the IPC. 

Various attempts have been made to cope with the prob- Although there have been known low-cost comniunica- 

lem above. Far example, according to a videophone appa- 30 t j on systems such as a videophone using analog 

ratus described in the JP-A-57- 129076, a background image transmission, to carry put transmission at a low transmission 

beforehand prepared is compared with a video currently rate, the picture quality has been sacrificed to a considerable 

being produced so as to accordingly clear the background, extent This consequently leads to the following problems, 

thereby achieving security control and nunirnizing the Expression appearing particularly in a human face cannot be 

amount of image information to be transmitted. 35 satisfactorily transmuted or displayed and variations in the 

However, when users of the system conduct communica- expression cannot be communicated in a realtime fashion, 

tion while viewing images of each other, the images of the resulting in an unnatural morion of the face, 

persons are most important in ordinary cases. Namely, Another object of the present invention is to provide a 

background images of the respective persons are less impor- videophone system capable of producing a high-quality 

tant in many cases. In consideration of effective allocation of 40 vide0 ^ a realtime manner even through such a 

the limited amount of codes, it can be consequently regarded transmission line having a low transmission rate as an 

as inefficient to uniformly encode the constituent elements ^ ^ thereby solving the above problem. 

of an image in an <^^^^^ n ^^. To achieve the above object according to the present 

assign the same quantity of codes to objects having different J ^ a 

v^es of szgruficance to &e commumcatmg users 45 ^ ^ of related to S a a 

^SS^** vtteophone apparato tecn^ in the ^deo camera mcluding extraction processing means for 

JP-A-57-r29<r76req^^ extracting ^ subject ^ot by mc camera and computing 

ground image in advance. Namely, conaderahon has not feamres feereof ^ mcans {ar mzlynng the 

been given to operabihty and usability of the users. fcaturcs from mc extraction processing means and convert- 

SUMMARY OF THE INVENTION ing the features into description of knowledge corresponding 

nistherefbreanobjectof the present invention to F ovide t» *c database, interface means for converting the descrip- 

a picture conmiiinication apparatus which can be used even *>n of knowledge generated by the encoding means into 

though a low-sptxrftransrnission path such as analog phone- signals cxmfonning to a signal system of a signal transmis- 

line while retaining a satisfactory quality of picture. 55 sion path and transmitting the knowledge description to a 

To achieve the object according to the present invention, r^ver and converting a signaksent from a sender into 
there are arranged image extraction means for extracting Ascription of knowledg^and decodu^ means for compos- 
images of r^ilar portions of an object, coding means for ^al according to the ^owledge description 
coding the «tracted^age portions and means for commu- * e ***** ade ^ reference to the database, 
nicating a partner with image data. 60 T^ constituent means above operate to achieve the object 

The image extraction means extracts images of particular 88 follows, 

portions of a subject Each of the extracted images is When the sender transmits an image, the video camera 
encoded in an encoding method or encoded by changing including, in addition to the extraction processing means and 
encoding parameters to produce a quantity of codes accord- encoding means, a signal processing circuit and a control 
ing to significance of the pertinent image portion. This 65 circuit commonly used for digital video cameras conducts 
optimally distributes codes to the respective portions of a signal processing known in the processing of video signals 

screen image. produced by an imaging apparatus to resultantry generate 
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such signals of the image as video signals. The extraction 
processing means extracts me subject from the video signal 
generated by the signal processing circuit to compute such 
features of elements of the extracted object as the size, 
contour, color, coordinates of center of mass, and gradient 
The encoding means including a micro computer or the like 
analyzes information of the features from the extraction 
processing means, recognizes elements constituting the 
object and states thereof, and transforms the recognized 
information items into knowledge description corresponding 
to the database including knowledge of models related to the 
subject The interface means transforms the knowledge 
description generated by the encoding means into signals 
conforming to a signal system of the transmission path and 
transmits the resultant signal through the transmission path. 

The signal received via the transmission path is converted 
by the interface means into knowledge description. Hie 
decoding means decodes the knowledge description to 
reconstruct the transmitted image. In this operation, the 
decoding means accesses the database keeping therein a 
large number of images of models related to the object and 
then selects therefrom video data items associated with the 
elements constituting the image sent from the sender so as 
to restore the original image. 

That is, the sender does not transmit the image itself. The 
image of an object such as a human face to be transmitted 
is beforehand transformed into knowledge description rep- 
resenting the image such that the knowledge description is 
sent as transmission data. In the receiver, the knowledge 
description is decoded into the image of the subject as 
above. 

With this provision, the amount of transmission data is 
remarkably minimized and hence it is possible to construct 
a videophone system capable of communicating high- 
quality pictures in a realtime manner even through such a 
communication line having a low transmission rate as an 
analog telephone line. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other objects and advantages of the present 
invention will become apparent by reference to the follow- 
ing description and accompanying drawings wherein: 

FIG. 1 is a diagram showing a first embodiment of an 
image communicating apparatus according to the present 
invention; 

FIG. 2 is a diagram showing the overall configuration of 
an image communicating apparatus according to the present 
invention; 

FIG. 3 is a diagram showing constitution of an image 
extracting circuit of the first embodiment; 

FIG. 4 is a diagram for explaining an encoding method of 
the first embodiment; 

FIG. 5 is a diagram for explaining the encoding method 
of the first embodiment; 

FIG. 6 is a diagram showing a second ernbodiment 
according to the present invention; 

FIG. 7 is a diagram showing structure of encoding means 
of the second embodiment; 

FIG. 8 is a diagram showing a memory map of video data 
in a storage; 

FIG. 9 is a diagram for explaining an encoding method of 
the second embodiment; 

FIG. 10 is a diagram showing constitution of a display of 
the second embodiment; 
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FIGS. 11A to 11C are diagrams for explaining an image 
model of human mouth; 

FIG. 12 is a diagram showing a third ernbodiment accord- 
ing to the present invention; 
5 FIG. 13 is a diagram showing a memory map of video 
data in a memory; 

FIG. 14 is a diagram showing the encoding method of the 
third embodiment; 
io FIGS. 15A and 15B are diagrams for explaining an 
example of the method of converting an image into knowl- 
edge description; 

FIG. 16 is a diagram showing an example of the contents 
of a database; 

13 FIG. 17 is a diagram showing an example of knowledge 
description; 

FIG. 18 is a diagram showing an image on the receiver 
side immediately after the communication line is established 
between the sender and the receiver, and 
20 FIGS. 19A and 19B are diagrams for explaining a method 
of receiving an image. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

25 

Next, description will be given of an embodiment of an 
picture communication apparatus according to the present 
invention. 

FIG. 2 shows structure of a picture communication appa- 

30 ratus such as a videophone facility including a user 1 
conducting communication via the apparatus, a video input 
apparatus 2, voice input device (microphone) 3, a display 
device 4, codec 5, and a communication network 6. 
Hie user 1 of the communicating apparatus communicates 

35 via a communication network with a partner using a similar 
communicating apparatus at a remote place. The imaging 
device 2 shoots an image of the user 1 and then inputs a 
video signal of the image to the encoder 5. The microphone 
3 transforms the voice of the user 1 into a voice signal to be 

40 fed to the encoder 5. The encoder 5 encodes the video and 
voice signals into a signal of code (communication signal) 
conforming to the network 6 and then supplies the signal to 
the network 6. In addition to transmission of the communi- 
cation signal from (he user 1 to the network 6, the decoder 

43 6 conducts reception of a communication signal sent from 
the communicating partner 1 via the network 6 and then 
decodes the signal to restore the video and voice signals of 
the partner L The resultant video and voice signals of the 
communicating partner 1 are fed to the display 4 to be 

so presented as an image and a sound. 

FIG. 1 shows an example of structure of the encoding 
device 5 of FIG. 2. A including an input terminal 20, output 
terminals 21, 22, an input and output terminal 23, an input 
signal before an extracting process 25a, an input signal after 

55 the extracting process 256, encoding circuits 27a, 276, a 
multiplexer 28, a transmission 1 receive unit 29, demulti- 
plexer 30, decorder 31a, 31fr, and a signal synthesize unit 32. 
According to the present invention, voices are processed in 
the ordinary known method and hence description thereof 

GO will be avoided. The video signal of the user 1 produced 
from the imaging apparatus and microphone is received via 
the input terminal 20. The signal is encoded by encoding 
means on the sender side including the extracting circuit 24, 
encoder circuits 27a and 27b, and multiplexer 28. The 

65 encoded signal is transformed by the sending and receiving 
section 29 into a communication signal to be outputted via 
the input and output terminal 15 to the network. The sending 
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and receiving section 14 conducts the transmission and which these regions are overlapped with each other, thereby 
reception at die same time and receives via the input and finally extracting as the region of the object an area sur- 
output terminal 23 a communication signal containing an rounding the overlapped area- 
image and a voice from the communicating partner. The According to the data items from the extraction circuit 24, 
signal is decoded by decoding means including the separat- 5 the control circuit 26 delivers a control signal from the 
ing circuit 30, decoding circuits 31c and 31fc, and compos- output terminal 27 to regulate the direction or orientation 
ing circuit 32 to restore the image signal of the partner. The and ratio of magnification of the imaging apparatus 2. As a 
image signal is then delivered from the output terminal 22. result, the imaging apparatus 2 is desirably and automati- 
The video signal is sent to the display 4 to be represented as cally oriented to the user 1 to shoot an image having an 
the image of the partner. Although not shown, when the l0 appropriate size. The processing procedure is executed as 
image of the user 1 is to be displayed on the display 4 for necessary so that the imaging apparatus 2 automatically 
confirmation, it is only necessary mat a change-over opera- follows movement of the commimicating person in front 
tion is conducted in the sending and receiving section 29 to thereof. To adjust the orientation and magnification ratio of 
treat the transmission signal as a reception signal. the imaging apparatus 2, the apparatus 2 may be mechani- 
Alternatively, the input video signal need only be supplied l5 cally or electronically operated. In an imaging apparatus 
to the composing circuit 32 to be mixed with a received including imaging devices such as charge-coupled devices, 
image so as to compose an image to be presented on the the electronic adjustment of orientation above can be 
display 4. achieved by using CCDs including marginal pixels which 

On receiving the signal from the extracting circuit 24, the are other than those used for the output of the imaging 
control circuit 26 sends a control signal to the imaging 20 apparatus. In addition, the magnification ratio can be elec- 
apparatus to obtain an optimal input image. The input image tronically conducted by an operation generally called elcc- 
signal is first fed to the extracting circuit 24 to extract partial tronic zooming. In the encoding circuit 27b, for the decoding 
images of the object In this embodiment, the shooting operation to be achieved later, the size and position of the 
object is the user of the apparatus. The partial images include extracted image are encoded together with the extracted 
the eyes, mouth, etc of the user. Since the images of the eyes 2s image 

and mouth vary in contour thereof more frequently than In operation on the receiver side, the received signals are 
those of the other elements of the object, it is necessary to separated by the separating circuit 30 into codes of extracted 
allocate a larger quantity of information items thereto. The portions and those of the other portions. The separated codes 
extracted partial images (extraction signal 252?) and other are respectively decoded by the decoders 21a and 31b 
partial images (non-extraction signal 25a) are inputted 30 corresponding to the encoders I7a and Z7b y respectively, 
respectively to the encoders 27a and 21b for the encoding There are resultandy attained images of the extracted por- 
thereof. Although the encoding method is not limited, to tions and images of the non-extracted portions. These 
restore a picture of a higher quality for the extracted partial images are fed to the composing circuit 32 to produce an 
images, a greater number of codes are generated from the image according to information items of the sizes and 
encoder circuit 21b. For the encoder circuit 27a, there may 35 positions of the extracted images, 
be utilized any encoding methods ordinarily used for video- FIG. 3 shows an example of constitution of the extracting 
phones (reference is made to ITU-T Recommendation H. circuit 24 of FIG. 1. The extracting method is basically 
261, Video codes for audiovisual services at px64 kbits identical to that described in the JP-A-4-205070. In the 
(1993) and to "encoding technology for videophone and configuration^ reference numerals 40 and 43 designate input 
television conference" written in page 793 of the Journal of 40 terminals, a numeral 41 denotes memory means including 
Institute of Television Engineers of Japan, VoL 47 (1991), one-bit data for each input pixel and keeping therein results 
No. 7). The encoder 27b may be operated according to the of decision for extraction areas, a numeral 45 indicates 
encoding method such as an entropy encoding method memory means, a numeral 42 stands for a decision circuit, 
(reference is made to page 106 of "Fundamentals of Elec- a numeral 44 represents an address generating circuit, and 
tronic Imaging System" written by W. F. Schreiber and 45 numerals 46 and 47 designates output terminals. A video 
published from Springer-Verlag in 1993). signal is fed via the input terminal 40 to the decision circuit 

The multiplexer 28 multiplexes signals of codes produced 42. The input terminal 43 is employed to input therefrom an 
from the encoders 27a and 276 in the preceding stage and extracting condition for each extraction portion. In this 
sends the multiplexed signal to the sender and receiver situatiori, it is aUowed to specify levels of me luminance and 
section 29. The extracting circuit 24 conducts, in addition to so chroma signals as the extraction condition. A plurality of 
extraction of partial images, an operation to compute for condition items are set for each extraction portion. For 
each extracted portion the size, contour, and position of a example, for the portion of the mouth, a plurality of com- 
reference point or coordinates of center of mass of the binations of luminance and chroma signal levels are set as a 
extracted portion and then outputs the resultant data items to red portion of the lip and a white portion of teeth. Since the 
the controller 26. ss lip color alters between persons, the luminance and chroma 

To sense an object and to obtain features thereof, there signal levels have allowance ranges, respectively. The deci- 
may be adopted, for example, a method described in the sion circuit 42 decides image areas conforming to the 
JP-A-59-208983 in which features of an object is attained extracting conditions in an image received from the input 
from differences between images sampled at a fixed interval terminal 40. The memory means 41 stores therein the results 
of time. Alternatively, there may be utilized a method 60 of processing of the decision circuit 4, namely, extraction 
described in the JP-A-4-205070 in which portions of a video image areas for each frame. Furthermore, the memory 
signal satisfying a preset condition, for example, a condition means 41 stores the extraction image areas for each extrac- 
determlned according to a luminance signal and a color tion portion. The results of decision are inputted again to the 
difference signal is regarded as candidates of the object The decision circuit 42 to be utilized as a candidate of an 
current candidate region thus extracted is compared with a 65 extraction region for the next frame. That is, the decision 
region of the object obtained before a predetermined period circuit 42 produces an image region by slightly expanding 
of time and stored in storage means to deterrnine an area in the previous extraction area for each extraction portion kept 
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in the memory means 41 to make decision for each extrac- of the respective constituent components of the face. An 
tion portion in the produced region. The decision circuit 42 example of the solid image display has been described, for 

computes the size and position of the image for each example, in JP-A-5-27346 and JP-A-3-22753. In a flat-plane 

extraction portion to deliver the results from the output or two-dimensional display, it is only necessary that the sizes 

terminal 46. The data items of the size and position are s of the respective components approximately reflect those of 

employed to control the imaging apparatus. According to the * e actua ' components of the object in the imaging operation 

control operation, the image of the user's face can be created thereof. However, in a three-dimensional display, the posi- 

in a fixed contour The positional data item of each extrac- ? ons <* * e components are required to cmrectty reflect 

tion portion is delivered to the address generating circuit 44 ftose rf ^ conmonents of the send dispfcy. Fortius 

to generate an address in the memory means 41, thereby to P^fP 0 * 6 ,' P 05 ^ 003 , of ^ <*» ■»* ««* fe « 

storing the extraction portions in separate locations, respec- to fit ***** *» resultantly decide the sizes of 

lively. HG. 4 shows an example of the results of address me elements in an automatic fashion. In this 

generation in which video data of extraction portions are co/inecbon, extracting circuit 125 computes positional 

combined with each other to configure one frame such as a relationships between the elements to attain the positions of 

OF (Common Interchange Format) for transmission. Image is ^ eves and dmas & * e Acting operation. In the 

data stored in the memory means 45 is later read therefrom P°f^°f adjustment^ the conttol circuit 105 is operated 

to be delivered from the output terminal 47. according to the positions obtained by the extracting circuit 

_ . . t . , , .. , . ...... 125 to adjust the imaging position and magnification ratio on 

FIG. 7 shows ameUiod of rnritiple^g twj^ds of video theil ^ sl ^ tas ^T ilea ^ s ^ woa:ssisappr> 

dafc .temsinch^ngdjose of toe exacted porbon (code A) priilt ^ r ^r out by between the verticaland 

and non-extracted portion (code B). The muhiplexmg of * hori20ntal f ^ b £ s<ycallttleitclIQoic ZOQm _ 

T± T 1°*^ tJt w *? ingfunction.The image of the face is regulated on the sender 

extracted portion, there is transnntted thelatest data for each side to ^tch the solid image display 90; thereafter, the 

frame; whereas, fortfae non-extracted portion one image a constituent components are extracted. 

transmitted in an interval of a predetermined number of , .. . . . . . . 

frames. Moreover, since the exacted portion includes a 25 ^^f^ circuit 125 is configured m the fame way 
partial image, when the image is decoded later by the f for mat . of &e P^^g embodiment shown in FIG. 3. 
j j .l . . - . jr c . t_ / . However, in case where the extracted face includes a phi- 
decoder, there is required information of the reproducing _ . \ , ; "r mv*u«wo « yxu 
rw*„...«ti„ *k. a :„„u.a~ „i« i^lity of constituent elements or variable number of con- 

Can f q "? f*>** A 11101114165 3150 P 05 * 00 * 1 stituent elements, it is difficult to combine the extracted 

information of each extracted image. , ' ^ [ " " " \_ , 7~ 

6 30 elements to match the format of the transmission frame as 

According to the embodiment, a greater quantity of codes described above (FIG. 4). Consequently, there will be intro- 

can be allocated to such images having a larger amount of duced another layout of the memory means 45 as shown in 

information as images related to the mouth and eyes. mG 8 This includes the respective constituent elements 

Resultantly, the quantity of overall transmission data nec- sirnp i y ^ mcm ory blocks of the predetermined sizes. The 

essary for achieving a^tisf actary quality of image can be ^ itcms outouttcd from ^ extracting circuit 125 can be 

reduced or an image of a higher quality can be obtained attained by issuing a read command to the memory means 

without increasing the amount of transmission data. 45, NamelVj me obtained data items constituting an image of 

FIG. 6 shows an alternative embodiment according to the each extracted portion, 

present invention. When compared with the preceding The encoder 127 encodes the output from the extracting 

embodiment, this embodiment includes a solid image dis- w circuit 125. The encoding method or various parameters 

play of the human head in place of the display of the use d in the encoding of data are determined, as described in 

preceding embodiment. A reference numeral 90 stands far conjunction with the preceding embodiment, according to 

the solid image display and a numeral 91 indicates an the kind and the priority levels for each extracted image, 

encoding apparatus. Each of me encoded images is multiplexed by the multi- 

FIG. 7 shows constitution of an encoder circuit 12 includ- 45 plexer 128 according to the priority level thereof, 

ing an input terminal 120, output terminals 121, 122, an FIG. 9 shows an example of the multiplexing process, 

input terminal 123, an input and output terminal 124, an Each frame includes a header code field containing, for 

extraction processing circuit 125, a control circuit 126, an example, information indicating an internal format of the 

encoding circuit 127, a multiplexer circuit 128, a sender and perdu cut frame and a frame identifier (ID) to identify the 

receiver section 129, a separating circuit 130, a decoder 50 frame and a video data field of each extracted portion. Video 

circuit 131, and a composing circuit 132. Functions of the data items of the respective constituent elements are distrib- 

respective circuit blocks are the same as the corresponding uted according to the priority levels so that the related 

constituent components of the preceding enibodiment The portions of a predetermined number of frames constitute 

extracting circuit 125 extracts the elements of the face and information of one frame. According to the distribution 

the entire face to present the human face on the display 90. 55 method of FIG. 9, only the portions of the face having a 

Assume that the constituent elements to be extracted are the lower priority level are distributed into a predetermined 

overall face, eyes, mouth, nose, eyebrows, etc. The elements number of frames for transmission thereof. The method of 

are assigned with priority levels for the encoding operation distributing codes into a plurality of frames requires a data 

thereof. For example, variation in the image of the entire buffering operation in the multiplexer 128. 

face is less than that in the image of the mouth and hence the ^ The multiplexed codes are transmitted via the sender and 

entire face is assigned with a lower priority leveL The eyes receiver section 129 to the network, on the receiver side, the 

and mouth are equally important in this regard and accord- codes are processed primarily by the demultiplexer circuit 

ingly assigned with the same priority level so as to allocate 130, decoder circuit 131, and synthesize unit 131 The 

a larger quantity of codes thereto. demultiplexer circuit 121 separates codes for each priority 

When it is impossible to vary the shape of solid image 65 level from the multiplexed codes from the sender side. The 

display according to the face, the display 90 is subjected to separated codes are respectively decoded by the decoder 131 

a model of a face having average features to fix the positions to reconstruct images of the respective constituent elements. 
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Since the frame frequency varies between video data items tion of variation or deformation thereof (variation 

received for the respective priority levels, the decoder 131 information). For the mouth image, the basic information 

includes memory means for updating video data in the includes lip image data as shown in FIG. 11B and the 

memory for each constituent element Hie internal memory variation information can be specified by the (jpening show- 
means may be configured in the niembry format of the 5 ^ ^^^* e U PP? m * ^ ^ 

~~rl. J . . & . o o^oir ~->a f™™ tfe*> as shown in FIG. 11C Video data of a variation of tbe mouth 

extracting circuit shown m FIG. 8. Signals read from the ~^ 7* . . . 

tL « ,„~a „, .icnXfrn^ ^^ri«- can be reconstructed by modifying the basic information 

memory means are used as output signals rrom tne decoder . . * . _ J % 

TT'^T^ according to the variation informaUon. Similar processing 

arcuit 131, also applies to the other extracted portions. 

On receiving the video output from the decoder 131, the An image supplied via the input terminal 120 is fed to the 

synthesize unit 132 composes an image of each constituent 10 extraction circuit 125 and undergoes an extracting operation, 

element Since the positions of the respective elements fiasic information obtained as a result of extraction is stored 

conform to information from the display 90, it is unneces- in toe memory means 180a and extracted images changing 

sary for the sender side to transmit positional information m a ^^^5 manner are supplied to the displacement 

together with the video data However, if the format for the calculating circuit 18L The basic information for mouth 

display 90 is unique, the display 90 need not transmit even 15 portion can be decided in two methods. In a first method, a 

the information above. point of time to acquire basic information is specified by (he 

FIG. 10 shows structure of the solid image display 90 user. In a second method, basic information is obtained by 

including a display unit thereof 140, a projecting unit 141 for the apparatus. In an example of the mouth, information 

projecting a picture onto the display unit 140, memory ^ related to an image of the mouth in an ordinarily closed state 

means 142, and an input and output (I/O) terminal 143. The assumed as basic information, whereas magnitude of 

memory means 142 stores therein data related to formats of variation thereof is used as variation information. In the first 

the display unit 140. The data includes data representing method, a point of time when the mouth image in the closed 

three-dimensional positions and sizes of the constituent state is obtained is determined by the user. In the second 

elements of the face. Since these data items are not changed method, an image of only the mouth is attained by an 

if the shape of the display is not variable, there is only extracting operation. Magnitude of opening of the mouth is 

required a read-only memory fixed for the display. Infer- monitored after the communication is started or during a 

mation of formats is sent via the I/O terminal 143 to the fixed period of time beginning at a predetermined point of 

encoding apparatus such that the encoder supplies the dis- time so as to decide a point of time when the magnitude 

play with an image conforming to the display. As above, if ^ takes a minimum value. This point of time is assumed to be 

information is communicated between the display and the when the mouth is closed, thereby attaining the basic infbr- 

encoding apparatus, it is possible to employ a display of mation. 

another type. The basic information for mouth portion attained by either 
When an image communication system includes the solid one of these methods is compared with the extracted image 
image display as above, there is obtained, in addition to the 35 at the specified point of time by the displacement calculating 
effect of the preceding embodiment, an advantageous fea- circuit 181 to obtain information of displacement The 
ture that the communication partner is reproduced in the encoding circuit 182 receives as inputs thereto the basic 
vicinity of the user in a three-dimensional manner. As a information and variation or displacement information and 
result, the communication can be achieved as if the partner then encodes the information. For each extracted image, the 
were in front of the user of the apparatus. Moreover, ^ obtained codes respectively of basic and variation informa- 
according to the embodiment, only the image of the human tion items are nwltiplexed by tbe multiplexer 128 to be 
face is transmitted, namely, the background image Is not transmitted via the sender and receiver section 129. 
included in the transmission data. Furthermore, the quantity fig. 13 shows the memory format of data items in the 
of codes are allocated for each portion according to the memory means 180a. Hie format may be similar to that of 
priority level or significance level thereof. Consequently, 45 me memory means in the extraction circuit 125. 
Mg^uahty pictures can be transmitted even through a nG 14 shows mc multiplexed data format employed by 
transmission path of a low transmission rate, the multiplexer 128. Basic information is trarisrnitted for 
FIG. 12 shows an alternative embodiment according to each pr edetermined number of frames. A frame not contain- 
the present invention. The diagram specifically shows con- m g the basic information is used to send variation informa- 
stitution of the encoding apparatus in which the same ^ tion. Each of the basic and variation information items 
constituent components as those of the preceding embodi- includes data items of the respective extraction portions, 
meats are assigned with the same reference numerals. The mc data reception, received codes are disassembled by 
apparatus of FIG. 12 includes memory means 180a, 1805, a ^ e demultiplexer circuit 130 into codes of the respective 
displa ce ment evaluatar 131, an encoder circuit 182, and a extraction blocks. Moreover, the codes are classified into 
decoder circuit 183. 55 those of basic information and those of displacement infer- 
In mis embodiment, in addition to extract partial images mation. Each unit of separated information is decoded into 
of the face, mere is conducted an operation to encode data of basic or displacement information by the decoder 
information related to structure of each portion of the face. 183. The basic and displacement information items are then 
As described above, the human face includes a plurality of sent to the memory means 1805 and composing circuit 132, 
portions and each portion has its own structure. FIG. 11 60 respectively. The composing circuit 132 reads the basic 
shows an example of structure of the human face. information from the memory means 1805 to execute an 
The image of the mouth section is considered to include operation of transforming the basic information according to 
the upper lip, lower lip, upper teeth, lower teeth, and interior the displacement information to reproduce each extraction 
of mouth as shown in FIG. 11A. These images do not portion and then arranges the respective extraction portions 
basically vary for a person during communication thereof. 65 at the pertinent positions to compose an image. The com- 
Consequently, information thereof can be classified into posed image is delivered as an output image from the output 
basic image information (basic information) and informa- terrninal 122. 
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According to the embodiment described above, each tion of a constituent element changed prior to data 

extraction portion is disassembled into basic information transmission, the amount of transmission data can be much 

including basic image data and displacement information more decreased. 

including displacement data relative to the basic information To restore the original image from the knowledge 

so as to transmit the resultant codes. The basic information 5 description, images corresponding to element numbers of 

including a larger number of codes is not transmitted at each the knowledge description are read from the database to be 

frame. Namely, the basic information is transmitted at an combined with each other so as to compose the objective 

interval of a predetermined number of frames, whereas the image. When arranging each constituent elements on the 

displacement information including a lower number of screen, the position described as (0,0) for the element in the 

codes is contained in each frame to be transmitted. This to knowledge description is aligned at the central position of 

remarkably decreases the quantity of transmission codes. the screen. As described above, since the position indicates 

Next, description will be given of a process in which an the difference between the coordinates of center of mass of 

image attained by an imaging apparatus is transformed into the object and that of each element, position (0,0) stands for 

knowledge description for transmission and received video the center of mass of the object With this provision, there 

data including knowledge description is converted into an 13 can be achieved correction of positions so that the object 

original image by reference to a database containing knowl- continuously stands at the central posldon of the screen in 

edge description data. any situations. 

Specifically, when an image extracted by the extracting In the direction of depth in the screen, the respective 

circuit 24 is encoded by the encoder Z7b f the database 40 is images are presented with such a positional relationship that 

referenced to transform the extracted image into knowledge 20 the smaller items are arranged in the upper layers. Moreover, 

description. when colors of images of constituent elements such as the 

Furthermore, when receiving image data in the form of ^ and iris of each eye in the database are replaced with 

knowledge description, the decoder 316 accesses the data- expressed by the knowledge description, the restored 

base according to the knowledge description to thereby ^ image will become more similar to the original image on the 

decode the video data into the original image. In this sender side. 

operation, video data items corresponding to the respective As above, the image itself is not used as the transmission 
elements constituting the image transmitted from the sender data. The image of a transmission object (such as a human 
side are selectively read from the database including a face) is transformed into knowledge description represent- 
multiplicity of images of models associated with objects to ^ ing the image so as to send data of the knowledge descrip- 
be imaged. The selected video data items are combined with tion to the communicating partner. On the receiver side, the 
each other to restore the original video image. Next, descrip- original image of object is restored according to the received 
tion wul be specifically given of knowledge description. For knowledge description. In consequence, the amount of trans- 
methods of describing knowledge, reference is to be made, mission data is considerably rninimized and there can be 
for example, to Chapter 8 (pages 132 to 139) of "Intelligent M provided a videophone system capable of producing a 
Image Processing" written by Agui and Nagasaki and pub- high-quality picture in a realtime fashion even through such 
lished from Shokodo in 1994. a conimuni cation line having a low transfer rate as an analog 

An example of the method of transforming an image of a telephone line, 
human into knowledge description will be described by Additionally, it may also be possible in the data commu- 
ref erence to FIGS. 15A and 15B. FIG. 15 A shows an image 40 ni cation that important elements of the object are transmitted 
of an object obtained when shooting a person by an imaging in the form of knowledge description and the other elements 
apparatus. From this image, an image related to the person are transferred as video signals. In this operation, the knowl- 
is extracted to be disassembled into constituent elements edge description is transmitted in a realtime manner, 
such as the hairs, face, eyes, mouth, and body so as to obtain whereas image information of the overall screen is trans- 
features including the coordinates of center of mass, width, 45 mitted at a low transfer speed in the range of the transfer rate 
height, size, and color of each element There are also of the communication path. When transmitting, for example, 
acquired such features as the width the iris of each eye, the an image of a human face, images of the eyes and mouth 
width and height of the interior of mouth, and gradient important for communication are sent in a realtime fashion, 
values of eyes and eyebrows. These features are transform Furthermore, when the image of the object shot by the 
into data items respectively assigned with element numbers 50 imaging apparatus is extracted from the overall image of 
in association with the database as shown in FIG. 16. object by the extracting circuit and the images of the 

FIG. 17 shows an example of knowledge description- For re mainin g portions are replaced with one color, the trans- 
each element, one set of knowledge description items is mission data can be more efficiently compressed, 
specified in the form of (element number, color(r-y,b-y), However, since information of the entire screen is trans- 
position(Ax^y), size). In this expression, position(Ax^y) 55 mitted at a low transmission speed in the method above, only 
indicates the discrepancy between the coordinates of center the eyes and mouth are displayed on the screen as shown in 
of mass of the pertinent object and that of each element As FIG. 18 immediately after the communication line is estab- 
can be seen from FIG. 17, data items of knowledge descrip- lished. To overcome this difficulty, there may be prepared a 
tion of constituent elements of object are described imme- model image of the human head portion in the database 1. 
diatdy after a frame demarcation code. Assume that the 60 Immediately after the communication line becomes 
object includes, far example, ten constituent elements and available, the eyes and mouth are composed according to the 
each element such as the element number is represented by knowledge description received in a realtime manner such 
an eight-bit data item. The amount of data required for each that the images of eyes and mouth are combined with the 
frame resultantly becomes 480 bits. As above, the volume of model image so as to display the composed image on the 
transmission data can be remarkably reduced by converting 65 screen as shown in FIG. 19 A. As can be seen from FIG. 19B, 
an image into knowledge description. In addition, when the when the model image is replaced thereafter with images 
system is configured to transmit only the knowledge descrip- sequentially received from the sender side, there is continu- 
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ously displayed a natural image even Immediately after the 
communication line is connected. Namely, the presented 
image is gradually changed from the model image into the 
human image of the sender without causing any undesirable 
artificial expression, and hence the viewer can obtain a 
naturally reproduced image. 

As above, even when there is utilized a communication 
line of a low transmission rate such as an analog telephone 
line, the ftlffmwits of expression and the like of a human face 
essential for communication can be transmitted in a realtime 
fashion while transferring video data of the overall screen 
image. This leads to an advantageous effect similar to that of 
the embodiment shown in FIG. 1. 

While the present invention has been described with 
reference to the particular illustrative embodiments, it is not 
to be restricted by those embodiments but only by the 
appended claims. It is to be appreciated mat those skilled in 
the art can change or modify the embodiments without 
departing from the scope and spirit of the present invention. 

We claim: 

1. A picture communication apparatus, comp r i sing : 
imaging means; 
voice input means; 

extracting means for extracting at least one portion of an 
image of a subject from an image produced by said 
imaging means; 

encoding means for respectively encoding the image 
portion extracted by said extracting means and a voice 
inputted by said voice input means; 

communicating means for coinmunicating via a commu- 
nication network data obtained by encoding the image 
portion and the voice by said encoding means; 

decoding means for decoding data received from said 
communicating means and thereby restoring the 
extracted image portion and the voice; 
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synthesizing means far composing an image; 

a display having a surface including depressions and 
projections for displaying the image composed by said 
synthesizing means; 

a memory far storing information representing three- 
dimensional positions and sizes of constituent elements 
of the image to be displayed on the depressions and 
projections of said display; 
10 data input/output means for transferring said information 
from said memory to said synthesizing means, whereby 
said synthesizing means synthesizes the image accord- 
ing to the information received from said data input/ 
output means and the extracted image portion decoded 
15 by the decoding means to produce data representing the 
synthesized image which is coordinated to the depres- 
sions and projections of said display in accordance with 
said three-dimensional positions and sizes; and 

projection means responsive to said synthesized image 
20 data for projecting a synthesized Image onto said 
display. 

2. A picture communication apparatus according to claim 
1, wherein 

the image of the extracted portion is a part of a human 
25 face; and 

the depressions and the projections in the surface of the 
display have a general contour similar to a human face. 

3. A picture communication apparatus according to claim 
1, wherein the image of me extracted portion is apart of the 

30 human face of a user of a picture communication apparatus; 
and- 

the information stored in said memory represents the 
remaining non-extracted portion of the human face of 
35 said user. 

* ♦ * * * 
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